Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Handle errors in a middle of a stream #384

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

Handle errors in a middle of a stream #384

wants to merge 3 commits into from

Conversation

slvrtrn
Copy link
Contributor

@slvrtrn slvrtrn commented Jan 20, 2025

Summary

ResultSet implementation was adjusted given the changes from 24.11 and ClickHouse/ClickHouse#68800; formats like CSV will properly throw by default if an error occurs during the stream. JSONEachRow and other streamable formats require http_write_exception_in_output_format: 0 in clickhouse_settings, so ClickHouse breaks the output stream in a similar way.

Important: The http_write_exception_in_output_format default value is 1; if it is not modified, the data stream with formats like JSONEachRow will not be broken if an error occurs.

Here are some curl examples.

This will have exit code 0:

curl "http://localhost:8123" --data-binary "select throwIf(number = 3, 'There was an error in the stream!') AS _, randomPrintableASCII(20) AS str from system.numbers limit 30000 SETTINGS max_block_size=1 FORMAT JSONCompactEachRow" 
 
[0, ">w$zM[t3K(t#y(MhOB\/]"]
["Code: 395. DB::Exception: There was an error in the stream!: while executing 'FUNCTION throwIf(equals(__table1.number, 3_UInt8) :: 3, 'There was an error in the stream!'_String :: 2) -> throwIf(equals(__table1.number, 3_UInt8), 'There was an error in the stream!'_String) UInt8 : 0'. (FUNCTION_THROW_IF_VALUE_IS_NON_ZERO) (version 24.12.2.29 (official build))"]

But this will have exit code 18 (see http_write_exception_in_output_format=0 in the params):

curl "http://localhost:8123?http_write_exception_in_output_format=0" --data-binary "select throwIf(number = 3, 'There was an error in the stream!') AS _, randomPrintableASCII(20) AS str from system.numbers limit 30000 SETTINGS max_block_size=1 FORMAT JSONCompactEachRow"

[0, "n\\|Y>U`Qa)op\"}0hcblo"]
__exception__
Code: 395. DB::Exception: There was an error in the stream!: while executing 'FUNCTION throwIf(equals(__table1.number, 3_UInt8) :: 3, 'There was an error in the stream!'_String :: 2) -> throwIf(equals(__table1.number, 3_UInt8), 'There was an error in the stream!'_String) UInt8 : 0'. (FUNCTION_THROW_IF_VALUE_IS_NON_ZERO) (version 24.12.2.29 (official build))
curl: (18) transfer closed with outstanding read data remaining

Related to #332 and #378

Checklist

  • Unit and integration tests covering the common scenarios were added
  • A human-readable description of the changes was provided to include in CHANGELOG

mshustov
mshustov reviewed Jan 21, 2025
View reviewed changes
packages/client-node/src/result_set.ts
@@ -165,6 +183,19 @@ export class ResultSet<Format extends DataFormat | unknown>
}
callback()
},
// will be triggered if ClickHouse terminates the connection with an error while streaming
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Triggered by whom?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mshustov mshustov mshustov left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@slvrtrn @mshustov